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ABSTRACT 

Developments in computer and information technologies continue to give opportunities 
for designing advanced E-learning systems while entailing objective and technical 
evaluation methodologies. Design and development of E-learning systems require time- 
consuming and labor-intensive processes; therefore any decision about these systems 
and their analysis needs systematic and structured guidance to lead to better decisions. 
Multi-Criteria Decision Analysis (MCDA) techniques are applicable in instructional 
technology-related research areas as well as in other academic disciplines. In this study, 
a conceptual domain model and a decision activity framework is proposed for E-learning 
systems. Instructional, technological, and administrative decision domains are included 
in this model. Finally, an illustrative example is given to show that AHP is an effective 
MCDA method for E-learning-related decisions. 

Keywords: E-learning, Multi-Criteria Decision Analysis 

INTRODUCTION 

Humans have to make decisions to select or act on something for different purposes, 
ranging from simple to complex, conscious to unconscious. At one time or another, all 
individuals have wished that a difficult decision was easy, and that there was a simple 
and straightforward way to follow up. For example, in purchasing a utility, there are 
many factors to consider, such as price, flexibility, brand name, support of manufacturer, 
etc. Factors such as complexity, uncertainty, multiple objectives, and different 
perspectives in decision-making constitute the basic sources of difficulty (Clemen, 1996). 
In this type of multi-factor decision-making, a person may consider the various factors 
intuitively or subjectively, while feeling the need for a quantitative approach. Most of the 
decision problems include a number of factors requiring multi-factor evaluation 
processes. Therefore, a decision process and its analysis need systematic and structured 
guidance along with the necessary analytical tools that would lead to better decisions. 

Developments in computer and information technologies continue to give opportunities 
for designing advanced E-learning systems while entailing objective and technical 
evaluation methodologies. One of the difficult decisions that the educators sometimes 
have to make is deciding on an alternative for E-learning applications. Unqualified 
instructional software or E-learning practice waste time and resources, and they cannot 
create effective learning and knowledge transfer (Merrill L, 1996). Functions of the 
courseware, the application type of E-learning system chosen (web-based, computer- 
based, mobile learning), or the instructional software itself (simulation, tutorial, problem 
solving, instructional game etc) generally determine the underlying main evaluation 
criteria. There are a number of studies dealing with decision problems and the solutions 
brought to them in many academic disciplines, which are also using individual or 
integrated multi-criteria decision analysis (MCDA) techniques for a decision making 
process. 
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Some of these studies include instructional technology. E-learning and its related 
research areas, and they research or adopt one of these MCDA techniques to meet the 
requirements of instructional technology-related decisions. However, the literature 
review suggests that there is a need for a study using the MCDA techniques with in a 
high-level decision framework for E-Learning systems. 

METHODOLOGY 

The goal of this study is to propose a conceptual domain model and a decision activity 
framework for E-learning systems. The proposed model in this study serves the purposes 
of describing the important decision domains for E-Learning systems and developing a 
decision-activity algorithm that includes the E-learning project management. E-learning 
design and development, and the other processes. Furthermore, an illustrative example 
is given for selecting the most appropriate E-Learning system based on three quality 
criteria. 

E-LEARNING SYSTEMS 

Advances in computer and information technologies have improved the delivery 
mechanism of teaching materials via electronic media, and have enabled the design of 
learner-centered and self-paced learning environments while overcoming the obstacles 
of time and location. Although the review of research and studies about E-learning 
suggest that there is no exact and clear definition of E-learning, some of the important 
characteristics that are related to E-learning need to be highlighted. 

The most known and common aspect is the technology on which any E-learning system is 
dependent. Shih et al. (2008) say that "e-learning is the acquisition and use of 
knowledge distributed and facilitated primarily by electronic means". In the view of 
some researchers, internet, satellite and the other electronic communication 
technologies constitute the general framework of E-learning applications. 

According to Turvey (2010), technological and managerial definitions seem to portray 
the E-Learning as a simple delivery mechanism. Therefore, the instructional aspects, 
such as facilitation of communication between instructors and learners and the 
constructivist approaches in knowledge-creating mechanism, should be included in the 
definition criteria. 

E-learning systems are also thought to be comprised of different components, through 
which many of these systems are designed, and where formative and summative 
evaluation processes are required. One of these components may be the delivery or 
application type of the E-Learning system. Internet or web-based mediums, local or wide 
area networking and mobile communicational frameworks generally determine the 
infrastructure of any E-learning system. 

The choice of the delivery type of E-learning will also lead to another decision: It is the 
naming issue of the E-learning application, whether Computer-Based Training, Internet- 
Based Training or Web-Based Training; each has sometimes been used as synonymous to 
E-learning. 

The other component is the adoption of an instructional paradigm or the determination 
of the design criteria to comply with throughout the instructional design processes. The 
instructional software chosen for the e-learning system, which is related to the 
application type of E-learning, is distinct from the delivery type choice. 
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In the literature, there are many studies examining how the research variables 
pertaining to learners' performance and satisfaction are affected by the instructional 
software. In a traditional instructional design approach. E-learning systems are 
developed in a structured and linear fashion. 

Analysis, design/development, implementation, and evaluation phases are sharply 
marked out, where each phase is providing the required input to the next phase. The 
major drawback of this approach is its rigidity, in which the instructional and system 
refinements are processed in a linear fashion after or during the design steps. During 
these design and development phases, instructional technology-related decisions would 
necessitate using some of the MCDA methods. 

MULTI-CRITERIA DECISION ANALYSIS 

The definition of MCDA could be "taking the different alternatives or choices of 
something under consideration in order to make a decision based on a number of 
standards". The Cambridge Dictionary gives the definition of "criterion" as, "A standard 
by which a person judges, decides about or deals with something." Within the context of 
decision analysis, MCDA methods primarily aim at providing the necessary means for 
organizing complex and conflicting information about alternatives, so that the decision¬ 
makers can take all the criteria and factors into account. Belton and Stewart (2002) 
point out that MCDA is a process, "which makes explicit and manages subjectivity and it 
integrates objective measurement with value judgment". 

MCDA methods have been categorized in different ways. One way is making the 
distinction between multi-objective and multi-attribute decision-making methods 
(Mendoza et al., 2006). It is suggested that the main difference is the number of 
alternatives which the decision methods have been based on. 

The multi-attribute methods are designed for choosing the discrete alternatives, where 
as the multi-objective methods are suited for the infinite number of alternatives defined 
by a set of constraints. Clemen (1996) regards the multi-objective decision-making as 
"determining how best to trade off increased value on one objective for lower value on 
another". 

Ho et al. (2010) group the most prevalent MCDA approaches into two main categories in 
their study, as individual and integrated methods. They ground this study in a review of 
the literature, which includes the multi-criteria decision-making approaches for supplier 
evaluation and selection problems. Data envelopment analysis (DEA), linear 
programming, goal programming (GP), multi-objective programming, analytic hierarchy 
process (AHP), analytic network process (ANP), simple multi-attribute rating technique, 
and fuzzy set theory are involved in individual MCDA approaches, whereas the integrated 
AHP and DEA, integrated AHP and GP, integrated AHP and multi-objective programming, 
integrated fuzzy and AHP, and finally integrated ANP and multi-objective programming 
are in the integrated MCDA approaches. One of the best-known general classifications of 
MCDA methods is the one that Belton and Stewart (2002) made. They classify the 
methods into three categories: 

Value Measurement Models 

"Numerical scores are constructed in order to represent the degree to which one decision 
option may be preferred to another. Such scores are developed initially for each 
individual criterion, and are then synthesized in order to effect aggregation into higher- 
level preference models"; 
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Goal, Aspiration or Reference Level Models 

"Desirable or satisfactory levels of achievement are established for each criterion. The 
process then seeks to discover options, which are closest to achieving these desirable 
goals or aspirations"; 

Outranking Models 

"Alternative courses of action are compared pair wise, initially in terms of each criterion 
in order to identify the extent to which a preference for one over the other can be 
asserted. In aggregating such preference information across all relevant criteria, the 
model seeks to establish the strength of evidence favoring selection of one alternative 
over another", s long as the defined problem or the goal(s) of a decision problem would 
be the primary determinant of which MCDA method is most appropriate, it is also crucial 
that decision-makers feel comfortable with the selected MCDA models. A detailed 
analysis of the MCDA methods and their conceptual foundations is out of the scope of 
this study. 

However, a general outline of most known methods, which are also practically applicable 
to E-learning decision problems, should be helpful for understanding the decision activity 
algorithm of the proposed model in this study. 

Weighted Sum Model (WSM) is a simple and practical MDCA method that is dependent 
upon the measure of relative importance of criteria viewed by the decision-maker, which 
is generally called "weight" (Pomerol et al., 2000). Given that x alternatives and y 
decision criteria exist in a decision problem, the best alternative is determined by the 
aggregation of performance value of alternatives and the weights attributed to criteria. 
Goal programming (GP) is an optimization method used for minimizing the deviations 
from goals which belong to an objective that has probably conflicting measures with the 
other objectives' measures. The overall purpose of GP is to achieve a solution that 
minimizes these deviations, and it emphasizes trading off criteria (Chang et al., 2009). 
Data Envelopment Analysis (DEA) is defined as a nonparametric method by which a 
decision-making unit (DMU) is measured (Ray, 2004). Giannoulis et al. (2010) note that 
DEA is a ranking technique where normalization is not required. Their argument is that 
DMU and outputs & inputs could be replaced with alternatives and criteria respectively. 
Analytic Hierarchy Process (AHP) is one of the main MCDA methods for complex decision¬ 
making problems involving subjective judgment. AHP is especially beneficial when 
people have different perspectives and priorities and components of any decision are not 
easy to compare or quantify. Analytic Network Process (ANP) and AHP both have 
iterative pair-wise comparisons to obtain a ranking of alternatives given in a decision 
problem. 

The main difference between AHP and ANP is that the former structures the problem into 
a hierarchy including a goal, criteria and alternatives, and the latter structures the whole 
process as a network. 

E-LEARNING DECISION DOMAINS 

The Decision Domains 

It is appropriate to give the dictionary definitions before relating the "domain concept" 
to E-learning systems. Cambridge Advanced Learner's Dictionary defines domain as "an 
area of interest or an area over which a person has control". Longman Dictionary 
describes this area with activity, interest, or knowledge, especially one that a particular 
person, organization deals with. The definition in Webster's Dictionary is probably more 
relevant to our study, "the content of a particular field of knowledge plus the knowledge 
that you are interested in or are communicating about". 
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E-learning systems are thought to be comprised of three main decision domains, which 
also determine the criteria for E-learning systems: instructional, technological, and 
administrative decision domains (Figure: 1). 



Figure: 1 

E-learning Decision Domains 

The instructional decision domain primarily includes instructional requirements. Task and 
context analysis, online-course description, and instructional design parameters may be 
given as determining criteria for this domain. 

E-learning application type, development and deployment software architecture, hosting 
services, multimedia design, etc., constitute the technological domain. The 
administrative domain has two dimensions: online course management and project 
management. 

While the course management task can be regarded as a low-level administration, the 
project management-related activities wrap the other processes. Thus, different types of 
decision-makers, a.k.a. decision actors, may participate in its unique or mutual E- 
learning decision cases to achieve their goals. 

A Use Case Diagram in the Unified Modeling Language (UML) is a type of behavioral 
diagram that defines the requirements of a system as viewed by the actors; and it is 
especially very often referred in software engineering design and development 
processes. 

It describes the system's behavior under various conditions, shows a set of use cases and 
actors and their relationships while modeling the context of the system. 

A use case describes what a system does without specifying how it does it. Figure 2 
shows the general outline of the decision use cases of an E-learning system, and it 
depicts how the decision actors accomplish their goals through the use cases. 

The main E-learning decision use case is the outcome of the other group of use cases. 
Instructional, technological, and administrative use cases are included in the first group. 
E-learning analysis, design and development, implementation, and evaluation use cases 
are in the second group. 
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The last group is made up of project management-related use cases, which are project 
initiation, project planning, project execution, project closure, and project evaluation 
processes. 



Figure: 2. 

E-learning Use Cases with Decision Actors 

The uses cases related with instructional, technological, and administrative domains are 
depicted in elliptical shapes in Figure 2, and they organize the E-learning system's 
requirements, analysis and design processes. The "E-learning Decisions" use case is base 
use case, and it is the kernel of the system including the all sub use cases shown in 
Figure: 2. Its main sub use cases, which are the instructional, technological, and the 
administrative ones, have also the other included sub use cases. Project management 
uses cases (initiation, planning, execution, closure, and evaluation) belong to the 
"Administration Decisions". Instructional use cases (analysis, design & develop, 
implement, evaluate) are in the "Instructional Decisions". The actors, such as instructors, 
administrators, and MCDA experts, are drawn in stick figure shapes that are representing 
the information exchange with use cases. The decision actors of an E-learning system are 
expected to play different sets of decision roles when interacting with these use cases. 
For example; the instructor, learner, technology expert, and the instructional designer 
actors participate in the decisions of task analysis, course description, and design 
parameters. The administrator actor approaches to an E-learning system from a view 
point of project management discipline, and she or he primarily concerns with the 
project related decisions. 

The E-learning Decision Activities 

The activity diagrams are generally used for the modeling of business processes, 
activities, and work flows of systems at a high level. It represents the how data move 
around a system of activities that describe some computational procedures as well as 
decisions. There are some basic notations specific to activity diagrams. 
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Figure: 3 

The Decision Activities for an E-learning Project 


The occurring activities are represented by the rounded rectangles, decision points are 
represented by diamonds, while the black bars represent parallel activities Figure: 3 
shows the swim lanes that are related with the decision domains. Each of the swim lanes 
groups the processes and decisions of an E-learning decision domain controlled by a 
decision maker. 


The first swim lane includes the administrator domain, and it has the activities that are 
primarily related with the project management processes. From the project initiation 
view point, a feasibility study, forming structure, and deciding the objectives of the 
project are main domain tasks to be put forward. Subjects such as financial and other 
project resources, quality and acceptance criteria are the issues to be considered at the 
project planning phase. After forming a project plan. E-learning project activities and the 
tasks, which are addressed in the project plan, are executed in a series of management 
processes. Project documentations, post E-learning implementation review and the 
project evaluation activities take place at the final stage of the administrator domain. 
Project initiation-execution related issues of the administrator domain are the most 
probable activities of this domain requiring decision analysis approach, and they are 
processed in the MCDA expert domain. The second swim lane in Figure 3 is responsible 
for all decision analysis tasks. The first step is defining the E-learning project goal or 
objectives. Later at the second step, criteria and alternatives pertaining to E-learning 
systems are specified. on 4 













Determining the proper MCDA method comes after the specifications of criteria. Should 
we select E-learning application software as a project goal, a multi-attribute MCDA 
method could be used for choosing one from the discrete software alternatives? The 
main instructional design and development activities for E-learning systems are included 
in the third swim lane. Although, analysis, design, development and evaluation phases 
are marked out, each phase provides the input to the next development phase. 

THE ILLUSTRATIVE EXAMPLE USING AHP METHOD 

The illustrative example of this study which is also similar to that of Uysal (2010) puts 
the requirements of any E-learning system into three main groups, which are 
instructional, technological, and managerial requirements respectively. It is thought that 
the instructional design, technology, and the managerial functionalities determine the 
quality criteria for any E-learning system. During the E-learning design and development 
processes, the need for a quantitative evaluation approach to translate the requirements, 
subjective or objective expectations of stakeholders into the measured and quantified 
parameters will lead us to a well known MCDA method, AHP. AHP is a systematic multi¬ 
criteria evaluation method developed by Saaty (1980), and has found a wide range of 
place in many solutions of different problems. It enables individuals to structure complex 
problems in a form of hierarchy, and it addresses how to determine the relative 
importance of a set of alternatives in a multi-criteria decision-making environment. AHP 
helps decision makers to determine the various factors with their weights that are 
pointing out their importance and laying out the hierarchy of the decision. Its general 
execution steps are as follows: 

1 st Step: The decision maker starts the overall procedures by defining the 
problem and setting up the goal related with the problem. 

2 nd Step: She or he determines the criteria reflecting the experts' opinions, 
and later the hierarchy is structured and reviewed. 

3 rd Step: Iteratively and respectively; 

> The pair wise comparisons are made for each alternative, 

> Criteria weights are calculated, 

> Consistency is checked. 

4 th Step: Finally, the weights of criteria are aggregated and the weights are 
combined to rank the alternatives for selecting. 

The execution steps carried out for the quality-based selection of an E-learning system 
are shown in Figure: 4. 



Figure: 4 

The AHP Method for Selecting an E-learning System 
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Step-1: Defining the Goal 

It is supposed that the system alternatives are limited from many to three, as E- 
Learning-1, E-Learning-2 and E-Learning-3 for simplicity purposes in this study. The goal 
is to select the most appropriate E-Learning system based on the quality criteria. 

Step-2: Determining the Criteria 

It is possible to determine many main and sub-criteria for the multi-factor evaluation 
processes. However, these criteria are restricted to three, such as instructional quality, 
technological quality, and administrative quality for simplicity purposes. 

Step-3: Pair-Wise Comparisons 

The key issue for AHP is the making iterative pair-wise comparisons. Decision makers 
need to compare two different alternatives (E-learning system) based on a defined 
criterion using a scale ranging from 1 to 9. (1 is equally preferred, 2-Equally to 
moderately preferred, 3-Moderately preferred, 4-Moderately to strongly preferred, 5- 
Strongly preferred, 6- Strongly to very strongly preferred, 7- Very strongly preferred, 8- 
Very to extremely strongly preferred, and 9 is Extremely preferred) 

E-learning decision maker starts making comparisons by looking at the criterion 
"instructional quality", and then respectively compare E-Learning systems; E-Learning-1 
with E-Learning-2, E-Learning-1 with E-Learning-3, and finally E-Learning-2 with E- 
Learning-3 for scoring purposes. A pair-wise comparison matrix is to be constructed at 
the end of these procedures. This matrix reveals the preference for "instructional quality" 
concerning the three E-Learning system alternatives. 

Assuming that the decision maker is an expert on instructional technology, it is 
determined that the E-Learning-1 is moderately preferred to E-Learning-2, E-Learning-1 
is extremely preferred to E-Learning-3, and E-Learning-2 is strongly to very strongly 
preferred to E-Learning-3 (Table: 1). We place 1 from upper left corner to the lower right 
corner of the matrix since each E-Learning system is equally preferred to itself. If the E- 
Learning-1 is moderately preferred to E-Learning-2 and it is scored as 3, and then E- 
Learning-2 will naturally be preferred to E-Learning-1 with the score of 1/3. The lower 
left of this matrix is completed using the same logical approach (Table: 2). 


Table: 1 

Initial Comparisons 


Instructional 

E-l E-2 

E-3 

Quality 

E-l 

3 

9 

E-l 


6 

E-l 




Table: 2 


Completed Comparison Matrix 

Instructional 

E-l E-2 

E-3 

Quality 

E-l 

1 3 

9 

E-l 

1/3 1 

6 

E-l 

1/9 1/6 

1 
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Evaluations for the Criterion 

Evaluation procedures start aftermath of the pair-wise comparisons. E-learning decision 
maker executes the same evaluation steps iteratively in order that the relative 
importance of each criterion is determined clearly. Although each criterion needs to be 
handled individually and the results are combined with the next criterion calculations, 
decision maker will only focus on the criterion "instructional quality". Because of 
limitations of this study, the calculations of other criteria will be assumed that they are 
performed in the same manner. To start and make them easier, we convert matrix 
numbers to decimals, and then get the column totals (Table: 3). The numbers in the 
matrix are divided by their respective column totals and a normalized matrix is achieved 
once the column totals have been found (Tabl: 4). 


Table: 3 

Matrix Converted to Decimals 


Instructional Quality 

E-l 

E-2 

E-3 

E-l 

1 

3 

9 

E-l 

0.333 

1 

6 

E-l 

0.111 

0.167 

1 

Column Totals 

1.444 

4.167 

16.0 

Table: 4 

Matrix Divided by Column Totals 

Instructional Quality 

E-l 

E-2 

E-3 

E-l 

0.69 

2 

0.72 

0 

0.562 

E-l 

0.23 

0 

0.24 

0 

0.375 

E-l 

0.07 

7 

0.04 

0 

0.063 


The priorities for "instructional quality" of the three E-Learning systems are determined 
by finding the average of the rows from the matrix of numbers (Table: 5). 

Determining Consistency Ratio for the Criterion 

AHP regards the consistency as a cardinal consistency. As an example, if A is thought to 
be two times more important than B, and B is considered to be three times more 
important than C, then A should be six times more important than C. If the decision 
maker judges that A is less important than C, a judgmental error occurs and the 
prioritization matrix is accepted as inconsistent. Therefore, the Consistency Ratio (CR) is 
a value, which is indicating that how we are consistent with our answers. A higher ratio 
means that the decision maker is less consistent, whereas a lower one means she or he is 
more consistent 

Table: 5 

Averages of the Rows 


Instructional Quality 


Row 

Averages 


(0.692 + 0.720 + 0.562)/3 = 0.658 

(0.230 + 0.240 + 0.375)/3 = 0.282 

(0.077 + 0.040 + 0.063)/3 = 0.060 
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Table: 6 

Decision Matrix for Instructional Design 


Criterion 

E-l 

E-2 

E-3 

Delivery Type 

0.658 

0.282 

0.060 

Instructional Design 




Instructional Software 





In terms of numbers, if the ratio is 0.10 or less, the decision maker's answers are 
consistent. A consistency ratio with a value higher than 0.10 requires reevaluation of the 
responses, which are given for the original matrix of pair-wire comparisons. In general, 
the division of the Consistency Index (Cl) by the value of Random Index (RI) gives us 
the CR. The basic formulas needed for the calculations of the CR are: 

cr = — and ci = — (1) 

RI n-l 

n is the number of alternatives; RI is the index number obtained from the table with an 
entry value of n; Lambda (A) is achieved from the matrix operations of the Weighted Sum 
Vector and the Consistency Vector as follows: 



(0.658)(1) + (0.282)(3) +(0.060)(9) 


2.044 

The Weighted Sum Vector = 

(0.658)(0333) + (0.282)(1) + (0.060)(6) 

= 

0.860 


(0.658)(0111) + (0.282)(0.167)+ (0.060)(1) 


0.180 



2.042/0.658 


3.103 

The Consistency Vector = 

0.860/0.282 

0.180/0.060 

— 

3.051 

3.009 


(3) 


ci = 


X - n 
n-l 


, 3.103+3.051+3.009 . ... 

A —-— j .02)4 


ci= 


X-n 3.054-3 


n-l 


3-1 


= 0.027 


(4) 


CI 0 027 

cr = —=—— = 0.047 is obtained by the value found by equation (4) and the value from RI 

RI 0.58 

table (Table: 7) 


Table: 7 

Random Index Table 


Number of 
Alternatives (n) 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Random Index 
(RI) 

0.0 

0 

0.58 

0.90 

1.12 

1.24 

1.32 

1.41 

1.45 

1.51 


It is possible to say that the decision maker is relatively consistent with his responses by 
looking at the CR, which has the value of 0.047. As a result, CR supports our original 
assessments of pair-wise comparison matrix. 

Evaluations for the Other Criteria 

So far, we have completed the evaluations for the criterion instructional quality for all 
alternatives of E-learning systems. The same calculations could be easily made for the 
other criteria that are named as technological and administrative quality. Assuming that 
we have performed the same pair-wise calculations, we end up with the final comparison 
matrix (Table 8). The next step is the determining the criteria weights. 
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Rather than deciding them subjectively, the AHP is again an objective method used for 
finding the weights. Iterative calculation methods and the computations of each CR 
enable an E-learning decision maker to be sure that he is also consistent with his 
responses to criteria weights. Table: 9 shows the weights of the criteria, which are also 
calculated in the same manner. 


Table: 8 

Pair-wise Comparison Matrix 


Criterion 

E-l 

E-2 


E-3 


Instructional 

Quality 

0.658 

0.282 

0.750 

0.060 





Criterion 

Weights 





Instructional 

0.082 

0.682 

0.236 

Technological 

Quality 

0.087 

0.182 


Quality 

Technological 

Quality 

Administrative 

Quality 

Administrative 

Quality 

0.497 

0.398 


0.107 



Table: 10 


The Final Decision 

Alternatives 

Decision 

Result 

E-l 

0.231 

E-2 

0.228 

E-3 

0.542 


Step-4: Selecting the E-learning System 

Following the completion of the comparison and criteria weight matrixes (Table: 8 and), 
the last step is obtaining of the final decision matrix. 

It is found by a matrix multiplication including the comparison and the criteria weight 
matrixes that are shown in Table 10. 

It is clear that the E-Learning-3 received the highest ranking in these calculations, and it 
should be selected as the best E-Learning system in terms of quality criteria. 

CONCLUSIONS 

Although advances in computer and information technologies have contributed to E- 
learning systems in a variety of ways, there are also many factors influencing their 
effectiveness. The literature review suggests that these factors need to be grouped and 
processed in a systematic framework, especially when the quality issue comes in to the 
question. 

Additionally, determining the best E-learning system which satisfies all the required 
criteria is a difficult educational task. 
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This difficulty arises from the fact that the various systems may have different 
characteristics, and not only one would posses all the instructional, technical and other 
functionalities for a qualified E-learning system. Thus, the evaluation of any E-learning 
system is considered as a complex multi-attribute decision-making problem. 

The decisions on E-learning systems need systematic and structured guidance as well as 
necessary analytical tools that would lead to better decisions. In this study, a decision 
framework model for E-learning systems is proposed, and this model has associated the 
instructional, technological, and administrative quality criteria with the corresponding 
decision domains. Finally, an illustrative example is given to show that AHP is an 
effective MCDA method as well as it is a promising tool for E-learning-related decisions. 
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